Method and apparatus for increasing frame rate of an image stream using at least one higher frame rate image stream

ABSTRACT

An image processing system comprises an image processor configured to obtain a first image stream having a first frame rate and a second image stream having a second frame rate lower than the first frame rate, to recover additional frames for the second image stream based on existing frames of the first and second image streams, and to utilize the additional frames to provide an increased frame rate for the second image stream. Recovering additional frames for the second image stream based on existing frames of the first and second image streams illustratively comprises determining sets of one or more additional frames for insertion between respective pairs of consecutive existing frames in the second image stream in respective iterations.

FIELD

The field relates generally to image processing, and more particularly to processing of multiple image streams having different frame rates.

BACKGROUND

Image processing is important in a wide variety of different machine vision applications, and such processing may involve multiple images of different types, possibly including both two-dimensional (2D) images and three-dimensional (3D) images, obtained from a variety of different image sources. For example, 2D images are provided by image sources such as video cameras and 3D images are provided by depth imagers such as structured light (SL) cameras or time of flight (ToF) cameras. Conventional image processing techniques therefore often require the processing of image streams from multiple distinct sources. However, problems can arise when the different sources generate images at different frame rates.

SUMMARY

In one embodiment, an image processing system comprises an image processor configured to obtain a first image stream having a first frame rate and a second image stream having a second frame rate lower than the first frame rate, to recover additional frames for the second image stream based on existing frames of the first and second image streams, and to utilize the additional frames to provide an increased frame rate for the second image stream. By way of example only, recovering additional frames for the second image stream based on existing frames of the first and second image streams may illustratively comprise determining sets of one or more additional frames for insertion between respective pairs of consecutive existing frames in the second image stream.

Other embodiments of the invention include but are not limited to methods, apparatus, systems, processing devices, integrated circuits, and computer-readable storage media having computer program code embodied therein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an image processing system in one embodiment.

FIG. 2 illustrates insertion of recovered additional frames into a low frame rate image stream so as to increase a frame rate of that image stream in the FIG. 1 system.

FIG. 3 is a flow diagram of an exemplary process for increasing a frame rate of a low frame rate image stream in the FIG. 1 system.

FIG. 4 is a flow diagram showing another exemplary process for increasing a frame rate of a low frame rate image stream in the FIG. 1 system.

FIGS. 5 through 7 are flow diagrams illustrating the operation of a cluster correspondence module of an image processor in the FIG. 1 system.

DETAILED DESCRIPTION

Embodiments of the invention will be illustrated herein in conjunction with exemplary image processing systems that include image processors or other types of processing devices and implement techniques for increasing a frame rate of a depth image stream or other type of image stream using at least one higher rate image stream. It should be understood, however, that embodiments of the invention are more generally applicable to any image processing system or associated device or technique that involves processing multiple image streams having different frame rates.

FIG. 1 shows an image processing system 100 in an embodiment of the invention. The image processing system 100 comprises an image processor 102 that communicates over a network 104 with a plurality of processing devices 106. The image processor 102 receives at least two distinct image streams from respective image sources 107. More particularly, the image processor 102 receives from a high rate image source 107-1 a first image stream Fs₁ having a first frame rate, and receives from a low frame rate image source 107-2 a second image stream Fs₂ having a second frame rate lower than the first frame rate. The image sources 107 may be assumed without limitation to provide image streams taken from the same or similar viewpoints of a given scene, such that similar pixel clusters or other image segments may be present in both the high frame rate and low frame rate image streams. The terms “high” and “low” as applied to frame rates in this context are relative terms, and should not be construed as requiring any particular absolute frame rates.

By way of example, the high frame rate image source 107-1 may comprise a video camera or other video source providing a sequence of 2D images, and the low frame rate image source 107-1 may comprise a depth imager such as an SL camera or a ToF camera that provides a sequence of depth images.

A wide variety of other types of image sources generating multiple image streams may be used in other embodiments, including 2D imagers configured to generate 2D infrared images, gray scale images, color images or other types of 2D images, as well as 3D imagers such as SL and ToF cameras, in any combination.

The image sources 107 may illustratively comprise respective image sensors each of which generates a separate image stream. The sensors may be separately installed and arranged apart from one another or may comprise different portions of a unified sensor having a first set of sensor cells used to generate a first image stream and a second set of sensor cells used to generate a second image stream. The first and second image streams may therefore be obtained from respective distinct image sensors configured for imaging of a given scene. Such distinct image sensors are generally assumed herein to capture substantially the same scene, but at different frame rates.

Although the first and second image streams Fs₁ and Fs₂ in the FIG. 1 embodiment are supplied by separate image sources 107-1 and 107-2, in other embodiments multiple image streams of different frame rates may be supplied using only a single imager or other type of image source. Thus, for example, the first and second image sources 107-1 and 107-2 may represent different portions of a single multi-stream imager, such as different sensors as indicated above. A given image source as that term is broadly used herein may comprise a memory or other storage device that stores previously-captured image streams for retrieval by or on behalf of the image processor 102. As another example, an image source may comprise a server that provides one or more image streams to the image processor 102 for processing.

Also, although only two input image streams are processed to increase a frame rate of one of the image streams in FIG. 1, in other embodiments more than first and second image streams may be used. For example, three or more separate image streams each with a different frame rate may be processed in order to recover additional frames so as to increase the frame rate of at least one of the image streams.

In the present embodiment, the image processor 102 is configured to recover additional frames for the second image stream Fs₂ based on existing frames of the first and second image streams Fs₁ and Fs₂, and to utilize the additional frames to provide an increased frame rate for the second image stream Fs₂. For example, the image processor 102 may increase the frame rate for the second image stream until it is substantially equal to the frame rate of the first image stream, such that the first and second image streams may be more easily processed by one or more destination devices.

More particularly, the image processor 102 is configured to determine sets of one or more additional frames for insertion between respective pairs of consecutive existing frames in the second image stream. Each such determination may be viewed as an iteration of one of the exemplary processes to be described in conjunction with the flow diagrams of FIGS. 3 and 4. For a given such iteration, a particular set of one or more additional frames is determined for insertion between a corresponding one of the pairs of consecutive existing frames in the second image stream based on a plurality of corresponding existing frames in the first image stream and the corresponding pair of consecutive existing frames in the second image stream.

This technique is illustrated in FIG. 2, which shows one example of correspondence between existing frames of the first image stream Fs₁ and existing frames of the second image stream Fs₂. The second image stream in this example is assumed to be a depth stream comprising a stream of depth images. As the first image stream has a higher frame rate than the second image stream, the first image stream over the period of time illustrated in the figure includes more frames than the second image stream. The frames of the second image stream that are marked as unknown in the figure are examples of additional frames that are recovered for the second image stream Fs₂ by the image processor 102 in order to provide an increased frame rate for the second image stream Fs₂.

Two consecutive existing frames in the second image stream Fs₂ correspond to R+1 consecutive existing frames in the first image stream Fs₁, such that an N-th existing frame in the first image stream corresponds to an M-th existing frame in the second image stream, and an (N+R)-th existing frame in the first image stream corresponds to an (M+1)-th existing frame in the second image stream. Thus, for every R+1 consecutive existing frames in the first image stream in this example, there are only two consecutive existing frames in the second image sequence.

Corresponding frames in the first and second image streams Fs₁ and Fs₂ may be frames that are captured at substantially the same time instance by respective image sensors. However, the term “corresponding” in this context is intended to be more broadly construed, so as to encompass other types of temporal relations between frames of the first and second image streams.

In recovering the additional frames for the second image stream Fs₂, the image processor 102 determines R+1 additional frames for insertion between the M-th existing frame and the (M+1)-th existing frame in the second image stream. As will be described in greater detail below, the R+1 additional frames determined for insertion between the M-th existing frame and the (M+1)-th existing frame in the second image stream are determined based on the corresponding R+1 consecutive existing frames in the first image stream and the M-th and (M+1)-th existing frames in the second image stream. For example, the recovery of additional frames may involve identifying correspondence between portions of one or more of the existing frames of the first image stream and portions of one or more of the existing frames of the second image stream, and forming at least one of the additional frames utilizing image information from the identified portions. In some embodiments, these portions are more particularly referred to as “clusters.”

As is apparent from the above, the image processor 102 in the present embodiment is configured to generate a modified second image stream Fs₂′ that includes one or more additional frames within a given period of time and therefore has a higher frame rate than the original input second image stream Fs₂.

It should be understood that the particular arrangement shown in FIG. 2 is presented by way of illustrative example only, and other types of correspondence between frames of first and second image streams may exist in other embodiments. The disclosed techniques can be adapted in a straightforward manner to any multiple stream arrangement having a frame rate ratio

$\frac{{Fr}_{1}}{{Fr}_{2}} > 1$

where Fr₁ denotes the frame rate of the first image stream Fs₁ and Fr₂ denotes the frame rate of the second image stream Fs₂.

The image processor 102 as illustrated in FIG. 1 includes a frame capture module 108 configured to capture frames from the input first and second image sequences Fs₁ and Fs₂ for further processing. It should be noted that “frame capture” in this context generally refers to one manner of obtaining particular frames of the first and second image sequences for further processing by the image processor. This is distinct from the original capture of the images by associated image sensors.

The captured frames are processed to recover additional frames of the second image sequence in order to provide a higher frame rate for the second image sequence. As indicated above, the higher-rate second image sequence is also referred to herein as a modified second image sequence Fs₂′, because it includes recovered additional frames that were not part of the original input second image sequence.

The recovery of additional frames for the second image sequence is performed using affine transform module 110, clustering module 112, cluster correspondence module 114, depth filing module 116 and postprocessing module 118 of the image processor 102. The operation of these modules will be described in greater detail in conjunction with the flow diagrams of FIGS. 3 through 7. Although these exemplary modules are shown separately in the FIG. 1 embodiment, in other embodiments two or more of these modules may be combined into a lesser number of modules.

The modified second image stream Fs₂′ generated by the image processor 102 may be provided to one or more of the processing devices 106 over the network 104. The processing devices 106 may comprise, for example, computers, mobile phones, servers or storage devices, in any combination. One or more such devices also may include, for example, display screens or other user interfaces that are utilized to present images generated by the image processor 102. The processing devices 106 may therefore comprise a wide variety of different destination devices that receive processed image streams from the image processor 102 over the network 104, including by way of example at least one server or storage device that receives one or more processed image streams from the image processor 102.

Although shown as being separate from the processing devices 106 in the present embodiment, the image processor 102 may be at least partially combined with one or more of the processing devices. Thus, for example, the image processor 102 may be implemented at least in part using a given one of the processing devices 106. By way of example, a computer or mobile phone may be configured to incorporate the image processor 102 and possibly one or more of the image sources 107. The image sources 107 may therefore comprise cameras or other imagers associated with a computer, mobile phone or other processing device. It is therefore apparent that the image processor 102 may be at least partially combined with one or more image sources or image destinations on a common processing device.

The image processor 102 in the present embodiment is assumed to be implemented using at least one processing device and comprises a processor 120 coupled to a memory 122. The processor 120 executes software code stored in the memory 122 in order to control the performance of image processing operations. The image processor 102 also comprises a network interface 124 that supports communication over network 104.

The processor 120 may comprise, for example, a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor (DSP), or other similar processing device component, as well as other types and arrangements of image processing circuitry, in any combination.

The memory 122 stores software code for execution by the processor 120 in implementing portions of the functionality of image processor 102, such as portions of modules 110, 112, 114, 116 and 118. A given such memory that stores software code for execution by a corresponding processor is an example of what is more generally referred to herein as a computer-readable medium or other type of computer program product having computer program code embodied therein, and may comprise, for example, electronic memory such as random access memory (RAM) or read-only memory (ROM), magnetic memory, optical memory, or other types of storage devices in any combination. As indicated above, the processor may comprise portions or combinations of a microprocessor, ASIC₁ FPGA, CPU, ALU, DSP or other image processing circuitry.

It should also be appreciated that embodiments of the invention may be implemented in the form of integrated circuits. In a given such integrated circuit implementation, identical die are typically formed in a repeated pattern on a surface of a semiconductor wafer. Each die includes image processing circuitry as described herein, and may include other structures or circuits. The individual die are cut or diced from the wafer, then packaged as an integrated circuit. One skilled in the art would know how to dice wafers and package die to produce integrated circuits. Integrated circuits so manufactured are considered embodiments of the invention.

The particular configuration of image processing system 100 as shown in FIG. 1 is exemplary only, and the system 100 in other embodiments may include other elements in addition to or in place of those specifically shown, including one or more elements of a type commonly found in a conventional implementation of such a system.

For example, in some embodiments, the image processing system 100 is implemented as a video gaming system or other type of gesture-based system that processes image streams in order to recognize user gestures. The disclosed techniques can be similarly adapted for use in a wide variety of other systems requiring a gesture-based human-machine interface, and can also be applied to applications other than gesture recognition, such as machine vision systems in robotics and other industrial applications.

Referring now to FIG. 3, an exemplary process 300 for increasing the frame rate of the second image stream Fs₂ is shown. The process is assumed to be implemented by the image processor 102 using at least its modules 108, 110, 112, 114, 116 and 118. The process in this embodiment includes steps 302 through 312, which are repeated for each of multiple iterations. At each iteration, the process obtains R+3 frames, including R+1 frames from the first image stream Fs₁ and two frames from the second image stream Fs₂, and recovers R−1 additional frames for insertion between the two frames of the second image stream Fs₂ in order to increase the frame rate of that image stream. The additional frames recovered by the process are also referred to herein as “connecting” frames of the second image stream. In addition, the frames in this embodiment are referred to as respective images. The term “frame” as used herein is intended to be broadly construed, so as to encompass an image or other types and arrangements of image information.

In step 302, the image processor 102 in a given iteration of the FIG. 3 process “gets” or otherwise obtains particular designated images from the Fs₁ and Fs₂ image streams. These images include R+1 images denoted Im₁(N) through Im₁(N+R) from the first image stream Fs₁ and two images denoted Im₂(M) and Im₂(M+1) from the second image stream Fs₂. It is assumed for clarity and simplicity of description that that all of the images Im_(i)(j) have the same size or resolution in pixels, although images of different sizes or resolutions may be used in other embodiments.

The R+3 images referred to in the context of step 302 are more generally referred to herein as the above-noted R+1 frames from the first image stream Fs₁ and the two frames from the second image stream Fs₂. The images may be obtained from their respective image streams using the frame capture module 108.

In step 304, an affine transform is applied to at least a subset of the R+3 images obtained in step 302. For example, the affine transform may be applied only to the R+1 images from the first image stream Fs₁, or only to the two images from the second image stream Fs₂. The affine transform is an example of a type of calibrating transform used to substantially equalize the viewpoints of the images in order to facilitate subsequent clustering operations. The affine transform may be based on results of image sensor calibration performed at sensor manufacturing or setup, and is applied using the affine transform module 110 of the image processor 102. In embodiments in which the viewpoints of the images from the first and second image streams are already substantially equalized, for example, due to placement or arrangement of the image sensors for the first and second image streams, the affine transform operation may be eliminated.

In step 306, the R+3 images are segmented into clusters by application of a clustering operation to these images. This operation is implemented using the clustering module 112 of image processor 102. The clustering operation more particularly involves generating a separate cluster map for each of the R+3 images. More particularly, a separate cluster map is generated for each image in the set of input images {Im₁(N), Im₁(N+1), Im₁(N+R); Im₂(M), Im₂(M+1)}. The cluster maps for different images may have different characteristics, such as different numbers of clusters and different cluster numeration orders.

By way of example, a given cluster map Cm_(i)(j) for image Im_(i)(j) may be defined in the following manner. Assume that the set of all pixels from image Im_(i)(j) is segmented into non-intersecting subsets of pixels with each such subset representing a cluster. The cluster map in this case may be in the form of a matrix Cm_(i)(j) having the same size as image Im_(i)(j). Element (m,n) from Cm_(i)(j) corresponds to the index of a particular cluster of Im_(i)(j) to which the image pixel having coordinates (m,n) belongs. Other types of cluster maps may be used in other embodiments. The term “cluster map” as used herein is therefore intended to be broadly construed. For example, at least a portion of the cluster correspondence information generated by cluster correspondence module 114 may also be generated in the form of one or more cluster maps.

A variety of different clustering techniques may be used in implementing step 306. A detailed example of such a clustering technique based on statistical region merging (SRM) will be described elsewhere herein. Conventional aspects of SRM are disclosed in R. Nock and F. Nielsen, “Statistical region merging,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, No. 11, November 2004, which is incorporated by reference herein. The clustering techniques in this embodiment generally attempt to ensure that the boundaries of the identified clusters include significant boundaries of corresponding objects in the imaged scene even if those objects may be located different distances from the image sensors for the respective first and second image streams, or may appear in different colors or with other differing characteristics in the two image streams.

In step 308, cluster correspondence information is determined based on the cluster maps generated for the respective images in step 306. This determination is performed by the cluster correspondence module 114 of image processor 102. The cluster correspondence information is indicative of correspondence between clusters of the cluster maps generated in step 306, and as noted above may itself be expressed at least in part using other types of cluster maps, such as cluster maps denoted as Cg₁ and Cg₂ elsewhere herein.

As one illustration, the cluster correspondence operation in step 308 may receive as its inputs the cluster maps Cm₁(N), . . . , Cm₁(N+R); Cm₂(M), Cm₂(M+1) from step 306 and the set of input images {Im₁(N), Im₁(N+1), . . . , Im₁(N+R); Im₂(M), Im₂(M+1)}. Determining cluster correspondence in this case involves finding relationships between sets of clusters in different images, so that, for example, non-intersecting sets of clusters of one input image Im₁(j) from the first image stream correspond to non-intersecting sets of clusters of another input image Im₂(k) from the second image stream.

Thus, the cluster correspondence may identify a set of multiple clusters in Im₁(j) as corresponding to a different set of multiple clusters in Im₂(k), with both clusters being associated with the same imaged object or objects. The sets of clusters of Im₁(j) and Im₂(k) are assumed to cover substantially the entire set of image pixels of those respective images. Also, images Im₁(j) and Im₂(k) are assumed to be captured at substantially the same time by respective image sensors, such as frames N and M or frames N+R and M+1 in the FIG. 2 example. Additional details of exemplary cluster correspondence determinations will be described below in conjunction with FIGS. 5 through 7.

In step 310, a depth filling operation is applied using the cluster correspondence information obtained in step 308. This generally involves merging depth data from corresponding clusters in the images of the first and second image streams into one or more of the additional frames. More particularly, as indicated in the figure, in recovering frame N+i, at least one cluster from frame N+i of the first image stream Fs₁ is filled with depth data from corresponding clusters from frames M and M+1 of the second image stream Fs₂. The depth data filled into the recovered frames in the depth filling operation may be preprocessed in a manner to be described elsewhere herein. In other embodiments, image information other than or in addition to depth data may be used in recovering the additional frames based on cluster correspondence information.

As indicated at step 312, the depth filling operation of step 310 is repeated for i=1 to R−1 in order to recover the full set of R−1 additional frames for use in increasing the frame rate of the second image stream Fs₂. These depth filling operations are performed by the depth filling module 116 of the image processor 102.

Although not explicitly illustrated in the FIG. 3 embodiment, postprocessing operations may be applied after the depth filling operation of steps 310 and 312 in order to enhance the quality of the recovered additional frames. Such postprocessing operations are implemented using the postprocessing module 118 of the image processor 102. In other embodiments, the postprocessing may be eliminated, or at least partially combined with the depth filling operations. Also, other sets of additional or alternative operations may be used in implementing the process 300, and the particular operations shown in the figure should therefore be viewed as exemplary only.

Another exemplary process 400 for increasing the frame rate of the low frame rate image stream Fs₂ is shown in FIG. 4. This process as shown takes as its inputs R+1 frames 402 from the first image stream Fs₁ and two frames 404 from the second image stream Fs₂, and generates as its outputs R−1 recovered additional frames 406 of the second image stream Fs₂. In the figure, the R+1 frames 402 from the first image stream are denoted as frames N through N+R, the two frames 404 from the second image stream are denoted as frames M and M+1, and the R−1 output frames 406 are denoted as recovered frames 1 through R−1. Again, other numbers and types of input and output frames may be used in other embodiments.

The process 400 includes processing operations for affine transform 410, clustering 412, cluster correspondence 414, depth filling 416 and postprocessing 418, illustrated in the figure as respective blocks which are assumed to be implemented by the respective modules 110, 112, 114, 116 and 118 of image processor 102. As mentioned previously, these processing operations are exemplary only, and other embodiments may use additional or alternative sets of operations to increase the frame rate of an image stream.

The affine transform in the present embodiment is applied only to the R+1 frames 402 from the first image stream Fs₁. This is illustrated in the figure by affine transform blocks 410-1, 410-2, . . . 410-(R−1) which process respective input frames N, N+1, . . . N+R from the set of input frames 402. The input frames 404 which comprise the two frames M and M+1 of the second image stream Fs₂ are applied directly to clustering block 412 and depth filling block 416. The affine transformed frames at the output of the affine transform blocks 410-1, 410-2, . . . 410-(R−1) are also applied to clustering block 412 and depth filling block 416, as indicated in the figure. The clustering block 412 generates cluster maps in the manner previously described, and the cluster correspondence block 414 determines correspondence between clusters in frames from the first and second input streams. The resulting cluster correspondence information, which as noted above may include additional cluster maps, is provided to the depth filling block 416 which utilizes that information to recover the R−1 additional frames of the second image streams. Postprocessing is separately applied to each of these additional frames as indicated by postprocessing blocks 418-1, 418-2, . . . 418-(R−1). These postprocessing blocks 418 provide at their respective outputs the R−1 output frames 406.

As in the FIG. 3 embodiment, the processing operations of the FIG. 4 embodiment are repeated for each of a plurality of iterations, with each such iteration determining a set of additional frames for insertion between a given pair of consecutive existing frames of the second image stream Fs₂ in order to increase the frame rate of that image stream.

Exemplary affine transform operations performed by affine transform module 110 of image processor 102 will now be described in greater detail.

Let Im_(i)(N) be a captured image in the N-th frame of image stream Fs_(i) where i=1,2. An affine transform T may be determined such that T(Im₁(N)) is placed into the coordinate system of Im₂(N).

In an embodiment in which image sensors are fixed relative to one another and their collocation is known in advance, affine transform T may be defined at sensor manufacturing or setup as T₁=T(T₂), where T₁ is a vector basis of the i-th image sensor in 3D space and the affine transform T is an amplication matrix, where an amplication matrix generally provides a linear transformation that maps one set of vectors to another set of vectors. Accordingly, a given amplication matrix can be used, for example, to transform a vector basis of a first image sensor to a vector basis of a second image sensor, where the vector bases are considered in a single 3D space.

Key points associated with the affine transform T are expected to be in either 2D or 3D space depending on the type of image sensors used, and may be selected manually or by an automatic technique (e.g., using edge analysis). A suitable number of key points for many practical applications will be on the order of approximately 20, although other numbers of key points can be used in other embodiments.

The affine transform may be determined by solving an overdetermined system of linear equations using the least squares method, as will now be described. Let m denote the number of key points, define D_(xyz) as a 2×m or 3×m matrix containing coordinates for every m points from Im₁(N) written column-by-column, and define T_(xyz) as a 2×m or 3×m matrix containing corresponding m points from Im₂(N) written column-by-column. Let A and TR denote an affine transform matrix and an associated translation vector, respectively, that are optimal in a least mean squares sense. For the 3D case:

$\left\{ {{{{A \cdot \begin{pmatrix} {x_{1}^{1}y_{1}^{1}z_{1}^{1}} \\ \ldots \\ {x_{m}^{1}y_{m}^{1}z_{m}^{1}} \end{pmatrix}^{T}} + \overset{\_}{TR}} = \begin{pmatrix} {x_{1}^{2}y_{1}^{2}z_{1}^{2}} \\ \ldots \\ {x_{m}^{2}y_{m}^{2}z_{m}^{2}} \end{pmatrix}^{T}},{D_{xyz} = \begin{pmatrix} {x_{1}^{1}y_{1}^{1}z_{1}^{1}} \\ \ldots \\ {x_{m}^{1}y_{m}^{1}z_{m}^{1}} \end{pmatrix}^{T}},{T_{xyz} = \begin{pmatrix} {x_{1}^{2}y_{1}^{2}z_{1}^{2}} \\ \ldots \\ {x_{m}^{2}y_{m}^{2}z_{m}^{2}} \end{pmatrix}^{T}}} \right.$

where (x_(i) ^(j),y_(i) ^(j),z_(i) ^(j)) are coordinates of the i-th key point of Im_(j)(N) and matrix A and vector TR collectively define the affine transform:

T _(xyz) =A·D _(xyz) +TR,

Matrix A and vector TR can be found as a solution of the following optimization problem:

∥A·D_(xyz)+TR−T_(xyz)∥²→min.

Using element-wise notation A=(a_(ij)), where (i,j)=(1,1) . . . (3,3), and TR=(tr_(k)), where k=1 . . . 3, the solution of this optimization problem in the least mean squares sense is based on the following system of linear equations, which comprises a total of 12 variables and 12m equations:

dR/da _(ij)=0, i=1,2,3, j=1,2,3,

dR/dtr _(k)=0, k=1,2,3.

After affine transform parameters A and TR are determined in the manner described above, image Im₁(N) is transformed into the coordinate system of Im₂(N) as follows:

D _(1xyz) =A·D _(xyz) +TR.

As a result of application of the affine transform, (x,y) coordinates of pixels in the resulting transformed image D₁ are not always integers, but arc instead more generally rational numbers. These rational number coordinates can be mapped to a regular equidistant orthogonal integer lattice of Im₂(N) using techniques such as nearest neighbor or interpolation. Such a mapping provides an image Im₁(N) having the same resolution as Im₂(N), although with possibly vacant pixel positions (i.e., undefined data) as the mapping may leave some points in the regular lattice unfilled.

The foregoing is just one example of an affine transform that may be implemented by image processor 102, and other embodiments can use other types of transforms or techniques to align the coordinate systems of the image sensors used to generate the first and second image streams.

Exemplary clustering operations performed by clustering module 112 of image processor 102 will now be described in greater detail. It should initially be noted that the clustering module may implement several different clustering techniques that require different levels of computational resources and switch between those techniques based on the current computational load of the image processor 102.

As mentioned previously, clustering techniques suitable for use in the clustering module 112 may be based on statistical region merging or SRM. Such techniques are generally resistant to random noise and have moderate computational complexity as well as good quantitative error bounds. Also, the degree of segmentation can be regulated in a manner that allows computational requirements to be dynamically controlled.

In a more particular example of an SRM-based clustering technique, each pixel of an actual image Im_(i)(j) is represented by a family of independently distributed random variables relating to an optimal image Id_(i)(j), with the actual image Im_(i)(j) being considered a particular observation of the optimal image Id_(i)(j). The actual and optimal images Id_(i)(j) and Im_(i)(j) are each separated into optimal statistical regions using a homogeneity rule specifying that inside each statistical region pixels have the same expectation, and expectations of adjacent regions are different.

This exemplary SRM-based technique implements recursive merging using a specified merging predicate P. Consider an arbitrary image Im_(i)(j). Let each pixel of Im_(i)(j) be represented by Q random variables. Then merging predicate P for two arbitrary regions R₁,R₂ of Im_(i)(j) can be expressed as follows:

${P\left( {R_{1},R_{2}} \right)} = \left\{ {\begin{matrix} {{true},} & {{{if}{{R_{1} - R_{2}}}} \leq \sqrt{{b^{2}\left( R_{1} \right)} + {b^{2}\left( R_{2} \right)}}} \\ {{false},} & {otherwise} \end{matrix},{{{where}{b(R)}} = {G\sqrt{\frac{1}{2\; Q{R}} \cdot {\ln \left( {{R}^{R}/\delta} \right)}}}}} \right.$

where |R| denotes the number of pixels in region R, G denotes the maximum possible value of a given pixel of the current image (e.g., G=2¹² for an image from a Kinect image sensor), and δ is a positive value less than 1. Accordingly, |R₁−R₂| denotes the magnitude of the difference between the number of pixels in region R₁ and the number of pixels in region R₂. This technique merges regions R₁ and R₂ into a single cluster if P(R₁,R₂)=true.

The technique starts at the pixel level, with every pixel initially being considered an individual region. The order in which the merging of regions is tested against the predicate P follows an invariant A which indicates that when any test between two parts of two different regions occurs, that means all tests inside these two regions have previously occurred. This invariant A can be achieved using a function f(pix₁,pix₂)=|pix₁−pix₂|, where pix_(i) is an image pixel value.

The SRM-based technique then proceeds in the following manner. First, all possible pairs of pixels (pix₁,pix₂) are sorted in increasing order of function f(pix₁,pix₂)=|pix₁−pix₂|, and the resulting order is traversed only once. For any current pair of pixels (pix₁,pix₂) for which R(pix₁)≠R(pix₂), where R(pix) denotes the current region to which pix belongs, the test P(R(pix₁)),R(pix₂)) is performed and R(pix₁) and R(pix₂) are merged if and only if the test returns true. At the completion of the merging process for the current image, the image pixels have been separated into multiple clusters with the clusters being characterized by a cluster map of the type of described previously.

The function f(pix₁,pix₂)=|pix₁−pix₂| is used in this embodiment as an approximation of the invariant A, although other functions can be used. Also, merging predicates and other parameters can be varied in the above-described SRM-based technique. Moreover, various clustering techniques not based on SRM may be used.

Exemplary cluster correspondence operations performed by cluster correspondence module 114 of image processor 102 will now be described in greater detail with reference to FIGS. 5 through 7. FIG. 5 illustrates an overall cluster correspondence process 500 which includes portions denoted as (1) and (2) that are illustrated in respective processes 600 and 700 of FIGS. 6 and 7. There is a single instance of portion (1) in step 502, and multiple separate instances of portion (2) in respective steps 504, 506, 508 and 510. Portion (1) generally involves determining correspondence between two cluster maps, and a given instance of portion (2) generally involves mapping of one set of clusters to another set of clusters. It is to be appreciated, however, that additional or alternative cluster correspondence operations may be used in other embodiments.

Referring initially to FIG. 5, the cluster correspondence process 500 includes steps 502 associated with portion (1) and steps 504, 506, 508 and 510 each associated with an instance of portion (2).

In step 502, correspondence is determined between cluster maps Cm₂(M) and Cm₂(M+1) and cluster maps Cg₂(M) and Cg₂(M+1) are formed.

In step 504, cluster map Cg₂(M) is mapped to cluster map Cm₁(N) to get cluster map Cg₁(N).

In step 506, cluster map Cg₂(M+1) is mapped to cluster map Cm₁(N+R) to get cluster map Cg₁(N+R).

In step 508, cluster map Cg₁(N) is mapped to cluster map Cm₁(N+1) to get cluster map Cg₁(N+1).

In step 510, cluster map Cg₁(N+R) is mapped to cluster map Cm₁(N+R−1) to get cluster map Cg₁(N+R−1).

The sequence of mapping operations illustrated by steps 504 and 508 continues for one or more remaining frames.

Similarly, the sequence of mapping operations illustrated by steps 506 and 510 continues for one or more remaining frames.

As indicated, process 500 upon completion of the above-noted sequences of mapping operations produces cluster maps Cg₁(N), Cg₁(N+1), . . . , Cg₁(N+R), Cg₂(M) and Cg₂(M+1) as well as the number of clusters k in each of these cluster maps.

The cluster maps Cg1 and Cg2 are also referred to as aligned cluster maps, and may be viewed as examples of what is more generally referred to herein as “cluster correspondence information.” Like other cluster maps disclosed herein, Cg1 and Cg2 cluster maps may be represented as respective matrices. Numerous other types of cluster correspondence information may be used in other embodiments of the invention.

The exemplary process 600 for performing portion (1) in step 502 of FIG. 5 will now be described with reference to FIG. 6. As noted above, portion (1) generally involves determining correspondence between two cluster maps. Process 600 for implementing this functionality includes steps 602 through 618.

Let C₁ and C₂ denote two cluster maps that are to be mapped to one another using the process 600 to be described. In the context of step 502 of FIG. 5, C₁=Cm₂(M) and C₂=Cm₂(M+1). Let N₁ denote the number of clusters in C₁ and let N₂ denote the number of clusters in C₂.

Consider an arbitrary cluster CL₁ from C₁ and an arbitrary cluster CL₂ from C₂. The cluster CL₂ is said to intersect with the cluster CL₁ if the following condition is met:

ρ(CL ₁ ,CL ₂)>threshold₁.

Here 0≦ρ(CL₁,CL₂)≦1 denotes a relative intersection measure of two sets of pixels. For example, a given one of the following symmetric and non-symmetric intersection measures may be used:

${\rho_{1}\left( {{CL}_{1},{CL}_{2}} \right)} = \frac{{{CL}_{1}\bigcap{CL}_{2}}}{{{CL}_{1}\bigcup{CL}_{2}}}$ and ${\rho_{2}\left( {{CL}_{1},{CL}_{2}} \right)} = \frac{{{CL}_{1}\bigcap{CL}_{2}}}{{CL}_{2}}$

The threshold value threshold₁ corresponds to a predefined threshold value (e.g., 0.1) which may be controlled as a parameter of the process.

Cluster mapping in the process 600 starts from empty cluster maps Cg₁ and Cg₂, which may be represented in the form of zero matrices. Three additional variables are initialized, including the number of clusters k, and variables Used₁ and Used₂ that denote sets of already used clusters from C₁ and C₂ respectively.

In step 602, a global initialization of the process sets Used₁={ }, Used₂={ }, Cg₁=0, Cg₂=0, and k=0.

In step 604, clusters from C₁ are sorted in order of decreasing size based on the number of pixels in each cluster.

In step 606, a determination is made as to whether or not all clusters from C₁ have been used for grouping into sets. If all the clusters from C₁ have been used, the process 600 postprocesses the clusters in Cg₁ and Cg₂ as indicated in step 607, and the process is then exited as indicated. If all the clusters from C₁ have not been used, the process 600 moves to step 608.

In step 608, an unused cluster CL₁ from C₁ is found.

In step 610, a set search initialization is performed by initializing set g₁ as {CL₁} and corresponding set g₂ as an empty set. The process then loops through steps 612, 614, 616 and 617.

In step 612, g₂ is defined as a set of clusters from CL₂ that intersect with clusters from g₁, as follows:

g ₂ ={CL ₂ ∈C ₂\Used₂ |∃cl∈g ₁:ρ(cl,CL ₂)>threshold₁}.

In step 614, ĝ₁ is defined as a set of clusters from CL₁ that intersect with clusters from the new g₂ defined in step 612, as follows:

ĝ ₁ ={CL ₁ ∈C ₁\Used₁ |∃cl∈g ₂:ρ(CL ₁ ,cl)>threshold ₁}.

In step 616, a determination is made as to whether or not g₁ is equal to ĝ₁ . If g₁ is not equal to ĝ₁, the process moves to step 617, and otherwise moves to step 618.

In step 617, g₁ is set equal to ĝ₁, and steps 612, 614 and 616 are repeated until the condition g₁=ĝ₁ is met, at which point the process moves to step 618. This condition will be met after a finite number of iterations of the loop comprising steps 612, 614, 616 and 617, as there are finite numbers of clusters in C₁ and C₂.

In step 618, k is increased by 1 and sets of used clusters are updated, such that k=k+1, Used₁=Used₁∪g₁, and Used₂=Used₂∪g₂. Also, g₁ and g₂ are added to the resulting cluster maps Cg₁ and Cg₂ by setting Cg₁(g₁) and Cg₂(g₂) both equal to k, where Cg₁(g₁)=k means that all elements of the matrix for cluster map Cg₁ corresponding to pixels from cluster g₁ are set to k and similarly Cg₂(g₂)=k means that all elements of the matrix for cluster map Cg₂ corresponding to pixels from cluster g₂ are set to k.

The process then returns to step 606 as indicated. As noted above, if it is determined in step 606 that all the clusters from C₁ have been used, the process 600 postprocesses the clusters in Cg₁ and Cg₂ as indicated in step 607. By way of example, such postprocessing, which is distinct from the postprocessing performed by postprocessing blocks 418 in FIG. 4, may involve identifying any clusters in Cg₁ that have corresponding empty clusters in Cg₂ and any clusters in Cg₂ that were not assigned to any of the clusters in Cg₁. In both of these cases, each of the identified clusters is combined into the neighboring cluster with which it shares the longest boundary, where the boundary length between two clusters is defined as the number of boundary pixels for these clusters, and where a given boundary pixel is a pixel that has one or more neighbors from both clusters.

The postprocessing in step 607 may involve additional operations using a depth distance measure between clusters. For example, such a depth distance measure may be defined as absolute difference between average depths for two clusters, or as absolute distance between mean depth of boundary pixels of two clusters. The additional operations may include merging neighboring clusters in Cg₁ if the corresponding depth distance measure between them is less than a predefined threshold threshold2. Portion (2) of the FIG. 5 process can be applied to determine the corresponding merging procedure for Cg₂.

Alternative techniques can be used in place of process 600. For example, it is possible to apply an exhaustive search of all cluster sets in both images M and M+1 of the second image stream.

The exemplary process 700 for performing a given instance of portion (2) in a particular one of steps 504, 506, 508 and 510 of FIG. 5 will now be described with reference to FIG. 7. As noted above, a given instance of portion (2) generally involves mapping of one set of clusters to another set of clusters, and may be viewed as a simplified version of process 600. In this simplified version, for example, clusters from Cg are kept unchanged, and only clusters from Cm are grouped into sets. Such an arrangement is particularly useful when the number of clusters in Cm is greater than the number of clusters in Cg. Process 700 for implementing this functionality includes steps 702 through 710.

In step 702, a global initialization of the process sets Used₁={ }, Used₂={ }, Cg₁=Cm₁, Cg₂=0, and k=0.

In step 704, a determination is made as to whether or not all clusters from C₁ have been used for grouping into sets. If all the clusters from C₁ have been used, the process 700 postprocesses the clusters in Cg₂ as indicated in step 705, and the process is then exited as indicated. If all the clusters from C₁ have not been used, the process 700 moves to step 706.

In step 706, an unused cluster g₁ from C₁ is found.

In step 708, g₂ is defined as a set of clusters from CL₂ that intersect with clusters from g₁, as follows:

g ₂ ={CL ₂ ∈C ₂\Used₂ |∃cl∈g ₁:ρ(cl,CL ₂)>threshold₁}.

In step 710, k is increased by 1 and sets of used clusters are updated, such that k=k+1, Used₁=Used₁∪g₁, and Used₂=Used₂∪g₂. Also, g₂ is added to resulting cluster map Cg₂ by setting Cg₂(g₂) equal to k, where Cg₂(g₂)=k means that all elements of the matrix for cluster map Cg₂ corresponding to pixels from cluster g₂ are set to k.

The process then returns to step 704 as indicated. As noted above, if it is determined in step 704 that all the clusters from C₁ have been used, the process 700 postprocesses the clusters in Cg₂ as indicated in step 705.

It is to be appreciated that the particular process steps used in the flow diagrams of FIGS. 3 through 7 are exemplary only, and other embodiments can utilize different types and arrangements of image processing operations. For example, the particular types of clustering operations can be varied in other embodiments. Also, steps indicated as being performed serially in a given flow diagram can be performed at least in part in parallel with one or more other steps in other embodiments.

Exemplary depth filling operations performed by depth filling module 116 of image processor 102 will now be described in greater detail.

After the cluster correspondence module 114 determines cluster correspondence information in the manner described above, that information is passed to the depth filling module 116 which recovers the additional frames for insertion between existing frames M and M+1 of the second images stream Fs₂. The depth filling module adds depth data from these existing frames to particular clusters in the additional frames.

By way of example, using the cluster correspondence information, the depth filling module can process Cm₁(N+i), i=1,2, . . . , R−1 to determine sets of clusters {CL₁, . . . , CL_(P)} and {CL₁′, . . . , CL_(T)′}, where P is less than the maximum number of different elements in Cm₂(M) from Im₂(M) and T is less than the maximum number of different elements in Cm₂(M+1) from Im₂(M+1).

As indicated previously, depth data may be preprocessed before being filled into the additional frames by the depth filling module 116. For example, depth data may be enlarged in the following manner:

${{dData}\left( {i,j} \right)} = {\max\left( {{\max\limits_{{cluster} \in {CSET}}\left\lbrack {\max\limits_{{{pix}{({i,j})}} \in {cluster}}\left( {{pix}\left( {i,j} \right)} \right)} \right\rbrack},{\max\limits_{{cluster} \in {CSET}^{\prime}}\left\lbrack {\max\limits_{{{pix}{({i,j})}} \in {cluster}}\left( {{pix}\left( {i,j} \right)} \right)} \right\rbrack}} \right)}$

where dData(i,j) denotes the depth data with coordinates (i,j) and pix(i,j) is a value of the image pixel with coordinates (i,j). Additionally or alternatively, the depth data may be smoothed as follows:

dData=smooth(dData, smooth_str)

where this smoothing technique replaces depth data dData(i,j) for a given pixel by a weighted sum of depth data for neighboring pixels. The neighborhood template size is given by smooth_str (e.g., smooth_str=5). It should be understood that these particular preprocessing operations are exemplary only, and other types of preprocessing prior to depth filling may be used in other embodiments.

After constructing depth data dData for a given cluster from Im₁(N+i), i=1,2, . . . R−1, the depth filling module 116 uses that depth data to fill a corresponding cluster in a given recovered image. As the given cluster in the example is extracted from an image of the high frame rate image stream Fs₁, it can provide much more accurate positioning and edges of an associated imaged object than the images of the low frame rate image stream Fs₂.

The recovered image may be expressed as a sum of indicators Ind over all clusters from cluster maps related to Im₁(N+i) as follows:

${{{iData}\left( {p,q} \right)} = {\sum\limits_{{CL} \in {{Cm}_{1}{({N + i})}}}^{\;}\; {{{Ind}\left( {{{pix}\left( {p,q} \right)} \in {CL}} \right)} \cdot {{dData}\left( {p,q} \right)}}}},$

where a given indicator Ind is given by:

${{Ind}\left( {x \in A} \right)} = \left\{ \begin{matrix} {1,} & {{{if}\mspace{14mu} x} \in A} \\ {0,} & {otherwise} \end{matrix} \right.$

and where p,q take on all values given by size(Im₂(M)). The recovered image of second image stream Fs₂ in this case is assumed to have the same size as the corresponding image of the first image stream Fs₁.

Exemplary postprocessing operations performed by postprocessing module 118 of image processor 102 will now be described in greater detail. These postprocessing operations are generally configured to enhance the quality of the recovered images, and may involve applying one or more filters to the recovered images to enhance sharpness or to eliminate undesirable redundant shadowing.

As a more particular example, redundant shadowing can be eliminated and overall recovered image quality enhanced by applying the following postprocessing operations. A given recovered image Im₂(N) is divided into non-intersecting clusters such that each shadow region pertains to a particular cluster. It is assumed that cluster CL from Im₂(N) did not change significantly if correlation between CL and its corresponding cluster CL′ from Im₂(N+1) is less or equal than a predefined threshold multiplied by the CL area. Accordingly, it is assumed that cluster CL did not change if:

$\frac{{sum}\left( {{sum}\left( {{C.}*C^{\prime}} \right)} \right)}{{sum}\left( {{sum}\left( {CC^{\prime}} \right)} \right)} > {change\_ thr}$

where change_thr denotes a predefined real value (e.g., 0.95), and C and C′ are matrices related to clusters CL and CL′ respectively. It is apparent that clusters that do not significantly change over time satisfy this condition.

Cluster CL from Im₂(N) is retained if it includes a shadow region. However, if corresponding clusters in Im₂(N+1) and Im₂(N+2) do not change for two consequent frames, depth data from CL is used to decrease shadow regions in Im₂(N+2). More particularly, let iData be a recovered image based on Im₂(N+2) clusters and let it contain shadow regions, and also let CL having matrix C be a cluster from Im₂(N) such that CL does not change significantly and related clusters for time N+2 contain shadow regions. The postprocessing identifies all (i,j) coordinates such that pix(i,j)=Inf, where pix(i,j) is a pixel from iData, and then for C(i,j)>0, the CL pixels of the identified (i,j) coordinates are assigned value m which is different than Inf, such that pix(i,j)=m. This forces iData to contain fewer shadow regions. The particular value used for Inf depends on the type of imager used to generate the corresponding image, as shadow regions may have different values for different imagers, and generally denotes any value that can be used to indicate a given pixel as being part of a shadow region. For example, it may be a large negative number.

This exemplary postprocessing incurs an additional processing delay of 1/Fr₂ seconds, or one frame delay, which is not perceptually significant and can therefore be accommodated in practical applications. Again, a wide variety of other types of preprocessing operations may be used in other embodiments.

Embodiments of the invention provide particular efficient techniques for increasing the frame rate of a lower frame rate image stream using at least one higher rate image stream. For example, the disclosed techniques allow the frame rate of an image stream to be increased even if the imaged scene includes a combination of static and moving objects that are not necessarily following linear trajectories. Such streams would otherwise be very difficult to process using conventional techniques such as motion interpolation.

It should again be emphasized that the embodiments of the invention as described herein are intended to be illustrative only. For example, other embodiments of the invention can be implemented utilizing a wide variety of different types and arrangements of image processing circuitry, modules and processing operations than those utilized in the particular embodiments described herein. In addition, the particular assumptions made herein in the context of describing certain embodiments need not apply in other embodiments. These and numerous other alternative embodiments within the scope of the following claims will be readily apparent to those skilled in the art. 

What is claimed is:
 1. A method comprising: obtaining a first image stream having a first frame rate and a second image stream having a second frame rate lower than the first frame rate; recovering additional frames for the second image stream based on existing frames of the first and second image streams; and utilizing the additional frames to provide an increased frame rate for the second image stream; wherein said obtaining, recovering and utilizing are implemented in at least one processing device comprising a processor coupled to a memory.
 2. The method of claim 1 wherein obtaining the first and second image streams comprises obtaining at least one of the first and second image streams as a depth image stream from a depth imager.
 3. The method of claim 1 wherein obtaining the first and second image streams comprises obtaining the first and second image streams from respective distinct image sensors configured for imaging of a given scene.
 4. The method of claim 1 wherein two consecutive existing frames in the second image stream correspond to R+1 consecutive existing frames in the first image stream, such that if an N-th existing frame in the first image stream corresponds to an M-th existing frame in the second image stream, an (N+R)-th existing frame in the first image stream corresponds to an (M+1)-th existing frame in the second image stream.
 5. The method of claim 4 wherein recovering additional frames for the second image stream based on existing frames of the first and second image streams comprises determining R+1 additional frames for insertion between the M-th existing frame and the (M+1)-th existing frame in the second image stream.
 6. The method of claim 5 wherein the R+1 additional frames determined for insertion between the M-th existing frame and the (M+1)-th existing frame in the second image stream are determined based on the corresponding R+1 consecutive existing frames in the first image stream and the M-th and (M+1)-th existing frames in the second image stream.
 7. The method of claim 1 wherein recovering additional frames for the second image stream based on existing frames of the first and second image streams comprises determining sets of one or more additional frames for insertion between respective pairs of consecutive existing frames in the second image stream.
 8. The method of claim 7 wherein for a given iteration a particular set of one or more additional frames is determined for insertion between a corresponding one of the pairs of consecutive existing frames in the second image stream based on a plurality of corresponding existing frames in the first image stream and said corresponding pair of consecutive existing frames in the second image stream.
 9. The method of claim 1 wherein recovering additional frames for the second image stream based on existing frames of the first and second image streams comprises: applying clustering operations to respective ones of the existing frames so as to generate respective cluster maps; and generating cluster correspondence information indicative of correspondence between clusters of the cluster maps.
 10. The method of claim 9 further comprising applying an affine transform to at least a subset of the existing frames of at least one of the first and second image streams prior to applying clustering operations to those frames.
 11. The method of claim 9 wherein the clustering operation applied to a given one of the existing frames comprises a clustering operation based on statistical region merging in which the given existing frame is separated into a plurality of clusters each corresponding to a different statistical region.
 12. The method of claim 11 wherein the clustering operation based on statistical region merging implements recursive merging using a specified merging predicate for two arbitrary statistical regions R₁ and R₂ of the given existing frame, in accordance with the following equation: ${P\left( {R_{1},R_{2}} \right)} = \left\{ {\begin{matrix} {{true},} & {{{if}\mspace{14mu} {{R_{1} - R_{2}}}} \leq \sqrt{{b^{2}\left( R_{1} \right)} + {b^{2}\left( R_{2} \right)}}} \\ {{false},} & {otherwise} \end{matrix},} \right.$ where |R₁−R₂| denotes the magnitude of the difference between the number of pixels in region R₁ and the number of pixels in region R₂, and b(R_(i)) is a function of the number of pixels in region R_(i) and a maximum possible value of a pixel in the given frame, such that regions R₁ and R₂ are merged into a single cluster if P(R₁,R₂)=true.
 13. The method of claim 9 further comprising utilizing the cluster correspondence information to perform a depth filling operation in which depth data associated with one or more clusters in one or more of the existing frames of the first and second image streams is added to corresponding clusters in one or more of the additional frames in conjunction with recovering said additional frames.
 14. The method of claim 1 wherein recovering additional frames for the second image stream based on existing frames of the first and second image streams comprises: identifying correspondence between portions of one or more of the existing frames of the first image stream and portions of one or more of the existing frames of the second image stream; and forming at least one of the additional frames utilizing image information from said identified portions.
 15. A computer-readable storage medium having computer program code embodied therein, wherein the computer program code when executed in the processing device causes the processing device to perform the method of claim
 1. 16. An apparatus comprising: at least one processing device comprising a processor coupled to a memory; wherein said at least one processing device is configured to obtain a first image stream having a first frame rate and a second image stream having a second frame rate lower than the first frame rate, to recover additional frames for the second image stream based on existing frames of the first and second image streams, and to utilize the additional frames to provide an increased frame rate for the second image stream.
 17. The apparatus of claim 16 wherein the processing device comprises an image processor, the image processor comprising: a clustering module configured to apply clustering operations to respective ones of the existing frames so as to generate respective cluster maps; a cluster correspondence module configured to generate cluster correspondence information indicative of correspondence between clusters of the cluster maps.
 18. The apparatus of claim 17 wherein the image processor further comprises a depth filling module, the depth filling module being configured to utilize the cluster correspondence information to perform a depth filling operation in which depth data associated with one or more clusters in one or more of the existing frames of the first and second image streams is added to corresponding clusters in one or more of the additional frames in conjunction with recovering said additional frames.
 19. An image processing system comprising: a first image source providing a first image stream having a first frame rate; a second image source providing a second image stream having a second frame rate lower than the first frame rate; and an image processor coupled to the first and second image sources; wherein the image processor is configured to recover additional frames for the second image stream based on existing frames of the first and second image streams, and to utilize the additional frames to provide an increased frame rate for the second image stream.
 20. The system of claim 19 wherein at least one of the first and second image sources comprises a depth imager. 